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Response to Arguments 

Applicants arguments with respect to claims 46, 48, 50 - 54, 56, and 58-60, filed 14 
June 2007, have been fully considered but they are not persuasive. 

Applicant's arguments with respect to claims 61 and 62, filed 14 June 2007, have been 
considered but are moot in view of the new ground(s) of rejection. 

Applicant argues that Kato et al. ("Marker Tracking and HMD Calibration for a Video- 
based Augmented Reality Conferencing System", 20 October 1999) deals with 2D object and 
does not have a solution for dealing with 3D objects (page 9 and 10) and cites that Kato et al are 
working on adding support for shared 3D objects (page 9). In Figure 4, Kato discloses a head 
that appears to be 3D, that looks 3D, that is 3D shaped, conforms with 3D shadows, and is a 3D 
model of a head (Figure 4). Kato discloses, "while using this system [AR conferencing system], 
users can easily change the arrangement of Virtual Monitors, placing the virtual images of 
remote participants about them in the real world and they can collaboratively interact with 2D 
and 3D information using a Virtual Shared Whiteboard (page 2, column 1, lines 10-14) and 
Augmented Reality is used to overlay virtual objects onto the real world (page 1, column 2, lines 
1 8 - 28). Kato also discloses a 3D image data model of a product in a pose corresponding to the 
pose of the whiteboard and ID cards (Figures 3 and 4, see also Section 4: Position and pose 
estimation of markers), which creates the desired visual perspective. 

Applicant argues that Harrington (US 6,898,307) does not disclose creation of an 
augmented reality video in which a 3D image of an object is augmented into a video stream 
(pages 10 and 1 1). Harrington discloses, synthetic image comprises a moving illustration or 
three-dimensional objects (column 4, lines 45 and 46) being employed as reference items 
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(column 4, lines 43 and 44) onto a piece of paper (column 4, line 40) or other templates (column 
4 5 line 44) that is sensed by a computer (column 4, line 39) wherein the computer interpret 
captured video frames (column 4, lines 39 and 40) and overlays the paper as seen by the user 
(column 4, lines 54 and 55) presented in the head mounted display (column 4, line 54) and 
overlay the application's image onto the paper (column 5, lines 35 and 36). 

Applicant argues that Harrington does not create a three dimensional image data model of 
a product in a pose corresponding to the pose of the moving model plane (page 12), Harrington 
discloses the user could select as the application image a particular internet browser engine 
which, based upon the located orientation of the piece of paper, would be transformed to exactly 
match the detected orientation of the piece of paper so that the displayed image provides the 
illusion of the web page actually appearing on the blank piece of paper being held by the user 
and once the position of the paper page is determined, information can be used to correctly 
distort the application's image so it overlays the paper in the eyes of the user (column 5, lines 4 - 
35). 

Applicant argues that Rhoads et al. (US 7,050,603) putting watermarks in video signals 
does not teach encoding hyperlink information into the augmented reality video. Rhoads 
discloses watermark may carry information or programmatic action or link to external 
information or an action, such as retrieval and output of information stored elsewhere in a 
database, website, etc (column 4, lines 31-39) and watermark can be embedded immediately 
into a video object layer after a graphical model is rendered to the video object layer (column 4, 
lines 40 - 49). It would have been obvious to embed a link to a website into a video or into a 
video object layer. 
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Applicant argues that Harrington does not disclose an animated 3D object, but admits that 
Harrington does disclose displaying a 3D object in different poses to highlight different 
perspectives of the 3D object, but it would make less sense for Harrington to disclose an object 
that is animated, therefore not obvious (page 12). The claim language recites, "the three 
dimensional image data model is an animation" (claim 62). Rhoads is relied upon to teach 
animated 3D objects. Harrington augments 3D objects in the real world through a video. 

The claim objection has been withdrawn due to applicant's amendment. 

The 35 U.S.C. 101 rejections have been withdrawn due to applicant's amendments. 

Claim Objections 

Claim 54 is objected to because there are two semi-colons in a row on line 15. 
Appropriate correction is required. 

Claim Rejections - 35 USC § 102 
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed publication in this 
or a foreign country, before the invention thereof by the applicant for a patent 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 
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Claims 46, 52 - 54, and 60 are rejected under 35 U.S.C. 102(a) as being anticipated by 
Kato et al. ("Marker Tracking and HMD Calibration for a Video-based Augmented Reality 
Conferencing System", 20 October 1999). 

Regarding independent claim 46, Kato discloses a 3D head in a back right corner of 
Figure 4 (Figure 4) and the resulting composite video image is displayed back to both eyes of the 
user (page 4, column 1, lines 16 and 17), which reads on the claimed editing a video stream to 
combine augmented virtual images with video data. Kato discloses an image with a whiteboard 
and ID cards that has markers (page 3, column 1, line 9 - column 2, line 4 and Figure 4), which 
reads on the claimed providing video data comprising images of a moving model plane having 
markers. Kato discloses calibrating the Head Mounted Display (HMD) and camera, and 
estimating an accurate position and pose of fiducial markers (page 4, column 2, lines 4-11), 
which reads on the claimed markers for calibrating a camera to track the motion of the model 
plane and the claimed determining a pose of the moving model plane according to the markers in 
the video data and calibration results. Kato discloses a 3D image data model of a product in a 
pose corresponding to the pose of the whiteboard and ID cards (Figures 3 and 4), which reads on 
the claimed creating a three dimensional image data model of a product in a pose corresponding 
to the pose of the moving model plane. Kato discloses the six markers are used to find the board 
orientation and align virtual images (page 6, column 1, line 6 - column 2, line 5), which reads on 
the claimed orientation of the markers determines the orientation of the three dimensional image 
data model. Kato discloses a transformation of the relationship between marker coordinates and 
camera coordinates (page 4, column 2, line 12 - page 6, column 1, line 3) and the results of the 
virtual images (Figures 3 and 4), which reads on the claimed determining an image 



Application/Control Number: 09/91 5,650 Page 6 

Art Unit: 2628 

correspondence between the camera calibration results associated with the moving model plane 
and the three dimensional image data model. Kato discloses the final result of the 3D virtual 
objects on the whiteboard and ID cards (Figures 3 and 4), which reads on the claimed rendering a 
3D model of the product based on the image correspondence and the claimed generating an 
augmented reality video by superimposing the rendered 3D model of the product on the moving 
model plane in the video data and the claimed displaying the augmented reality video. 

Regarding independent claim 54, claim 54 is similar in scope as to claim 46, thus the 
rejection for claim 46 hereinabove is applicable to claim 54. 

Regarding dependent claims 52 and 60, Kato discloses a human manipulating an ID card 
(Figure 3), which reads on the claimed video data includes images of a human manipulating the 
pose of the model plane, wherein a rendering of the three-dimensional image model is 
manipulated according to the pose of the model plane. 

Regarding dependent claim 53, Kato discloses the system uses computer vision 
techniques and have optimized algorithms for fast, accurate real time registration and convenient 
optical see-through HMD calibration (page, 2, column 1, lines 18 - 26), which reads on the 
claimed wherein the augmented reality video is produced in real time. 

Claim 61 is rejected under 35 U.S.C. 102(e) as being anticipated by Harrington (US 
6,898,307). 

Regarding independent claim 62, Harrington discloses a camera 12 that captures the 
normal field of view of the user of the headset 10 that is substantially what the user would see 
without the headset 10 (column 4, lines 4 - 24) and the analog signal from the camera 12 is 
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converted to a digitized frame that can be stored in the computer 1 8 for analysis (column 4, lines 
25-38) and the computer 1 8 must interpret the captured video frames to locate a piece of paper 
within the camera view (column 4, lines 38 - 67 and column 5, lines 1 - 34) and the preselected 
movable real object comprises a reference panel such as a screen, tablet or piece of paper and the 
identifying includes recognizing a corner of the panel (claim 2), which reads on the claimed 
providing video data comprising images of a moving model plane having markers. It is inherent 
that a piece a paper, screen, or tablet is substantially rectangular, which reads on the claimed 
model plane is a substantially rectangular plane. Harrington discloses marker is a corner of the 
panel (claim 2) and a piece of paper can be identified by its white color against a darker 
background (column 4, line 55 - column 5, line 3), which reads on the claimed marker is a 
graphic disposed on an upper surface thereof. Harrington further discloses the unprojection 
module 82 that projects three-dimension object onto a two-dimensional viewing plane, where a 
two dimensional image results from the three dimensional object (column 7, lines 28 - 53). 
Harrington further discloses the unprojection step comprises the mathematical calculation 
comprising taking a two-dimensional image and determining what threerdimensional object 
could have created that two-dimensional image with respect to what the object can be and where 
it could have been positioned and the subject invention utilizes known dimensions with regard to 
the camera system and viewing plane, in combination with assumed characteristics of the piece 
of paper (column 7, lines 28 - 53), which reads on the claimed determining a pose of the moving 
model plane according to the markers in the video data, the pose highlighting a particular 
perspective of the moving model plane and the claimed determining an image correspondence 
between the moving model plane having markers and the three-dimensional image data model. 
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Harrington discloses the camera provide the computer with the same view that the user sees so 
that the synthetic image is displayed or superimpose in a manner for comfortable and convenient 
interaction between the real reference item within the field of view that the user may be holding 
and manipulating (column 4, lines 4 - 37), which reads on the claimed producing the augmented 
reality video by rendering the three-dimensional image data model of the product superimposed 
on the moving model plane in the video data. Harrington discloses the user could select as the 
application image a particular internet browser engine which, based upon the located orientation 
of the piece of paper, would be transformed to exactly match the detected orientation of the piece 
of paper so that the displayed image provides the illusion of the web page actually appearing on 
the blank piece of paper being held by the user and once the position of the paper page is 
determined, information can be used to correctly distort the application's image so it overlays the 
paper in the eyes of the user (column 5, lines 4 - 35) and the unprojection step comprises the 
mathematical calculation comprising taking a two-dimensional image and determining what 
three-dimensional object could have created that two-dimensional image with respect to what the 
object can be and where it could have been positioned and the subject invention utilizes known 
dimensions with regard to the camera system and viewing plane, in combination with assumed 
characteristics of the piece of paper (column 7, lines 28 - 53), which reads on the claimed 
rendering of the three-dimensional image data model has substantially the pose of the moving 
model plane and the claimed creating a three dimensional image model of a product in a pose 
corresponding to the pose of the moving model plane, wherein orientation of the markers 
determines the orientation of the three dimensional image data model and the claimed display the 
augmented reality video. Harrington discloses video capture hardware that converts the analog 
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signal from the camera into a digitized frame that can be stored in the computer and the hardware 
outpus a signal to the computer as a digitized and processible representation of what the user sees 
from the camera's field of view and the video generation hardware takes a bit map of pixel 
values from the computer and converts them into a television format that can be displayed on the 
headset and where the signal generated by the camera is different from the signal displayed on 
the headset by the addition of the computer-generated electronic image to thereby form a 
synthetic image as an augmented-reality display (column 4, lines 25 - 38), which reads on the 
claimed converting the augmented reality video into a streaming video format and the claimed 
streaming the augmented reality video having the streaming video format over a communications 
network to a computer for displaying the augmented reality video. 

Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claims 48 and 56 are rejected under 35 U.S.C. 103(a) as being unpatentable over Kato et 
al. ("Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing 
System", 20 October 1999) in view of Harrington (US 6,898,307). 

Regarding dependent claims 48 and 56, Kato did not expressly disclose scaling the three- 
dimensional image data model to the model plane according to the markers, though Kato does 
disclose simple transformation that could include scaling and also, transformations of the 
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relationship between marker coordinates and camera coordinates (page 4, column 2, line 12 - 
page 6, column 1, line 3) and the sizes of the virtual screens on two ID cards being of different 
sizes and positions (Figure 3). Harrington discloses the translation, rotation, scaling and 
perspective operations can be applied to the image using standard image processing methods, or 
specialized textual mapping hardware may be employed for improving processing speed (column 
5, lines 26 - 35). It would have been obvious for one of ordinary skill in the art at the time of the 
invention to modify Kato's system to scale the three-dimensional virtual image according to the 
markers. One would be motivated to do so because simple known algebraic transformation in 3- 
D space helps improve processing speed and scaling the virtual images depending on the markers 
would prevent inaccurate and/or obscured virtual images. 

Claim 62 is rejected under 35 U.S.C. 103(a) as being unpatentable over Harrington (US 
6,898,307) in view of Rhoads et al. (US 7,050,603). 

Regarding independent claim 62, Harrington discloses a camera 12 that captures the 
normal field of view of the user of the headset 10 that is substantially what the user would see 
without the headset 10 (column 4, lines 4 - 24) and the analog signal from the camera 12 is 
converted to a digitized frame that can be stored in the computer 18 for analysis (column 4, lines 
25-38) and the computer 1 8 must interpret the captured video frames to locate a piece of paper 
within the camera view (column 4, lines 38 - 67 and column 5, lines 1 - 34) and the preselected 
movable real object comprises a reference panel such as a screen, tablet or piece of paper and the 
identifying includes recognizing a corner of the panel (claim 2), which reads on the claimed 
providing video data comprising images of a moving model plane having markers. It is inherent 



Application/Control Number: 09/91 5,650 Page 1 1 

Art Unit: 2628 

that a piece a paper, screen, or tablet is substantially rectangular, which reads on the claimed 
model plane is a substantially rectangular plane. Harrington discloses marker is a corner of the 
panel (claim 2) and a piece of paper can be identified by its white color against a darker 
background (column 4, line 55 - column 5, line 3), which reads on the claimed marker is a 
graphic disposed on an upper surface thereof. Harrington further discloses the unprojection 
module 82 that projects three-dimension object onto a two-dimensional viewing plane, where a 
two dimensional image results from the three dimensional object (column 7, lines 28 - 53). 
Harrington did not expressly disclose the three-dimensional image data model is an animation. 
Rhoads discloses embedding watermark in an animated 3D graphical objects and the embedded 
object can be composited with a video stream to form a video program, such as a movie or 
television program and graphical objects that link to information or electronic commerce 
transactions can be added to a video product, such as a movie, when its converted from one 
format to another (column 18, lines 5-18). It would have been obvious for one of ordinary skill 
in the art at the time of the invention to modify Harrington's system by projecting 3D animated 
objects. One would be motivated to do so because 3D animated objects are used in many 
commonly used applications such as games (column 18, line 63 - column 19, line 5) or 
interactive shopping (column 20, lines 41-48) that provide users a more convincing virtual 
environment. Harrington further discloses the unprojection step comprises the mathematical 
calculation comprising taking a two-dimensional image and determining what three-dimensional 
object could have created that two-dimensional image with respect to what the object can be and 
where it could have been positioned and the subject invention utilizes known dimensions with 
regard to the camera system and viewing plane, in combination with assumed characteristics of 
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the piece of paper (column 7, lines 28 - 53), which reads on the claimed determining a pose of 
the moving model plane according to the markers in the video data, the pose highlighting a 
particular perspective of the moving model plane and the claimed determining an image 
correspondence between the moving model plane having markers and the three-dimensional 
image data model. Harrington discloses the camera provide the computer with the same view 
that the user sees so that the synthetic image is displayed or superimpose in a manner for 
comfortable and convenient interaction between the real reference item within the field of view 
that the user may be holding and manipulating (column 4, lines 4-37), which reads on the 
claimed producing the augmented reality video by rendering the three-dimensional image data 
model of the product superimposed on the moving model plane in the video data. Harrington 
discloses the user could select as the application image a particular internet browser engine 
which, based upon the located orientation of the piece of paper, would be transformed to exactly 
match the detected orientation of the piece of paper so that the displayed image provides the 
illusion of the web page actually appearing on the blank piece of paper being held by the user 
and once the position of the paper page is determined, information can be used to correctly 
distort the application's image so it overlays the paper in the eyes of the user (column 5, lines 4 - 
35) and the unprojection step comprises the mathematical calculation comprising taking a two- 
dimensional image and determining what three-dimensional object could have created that two- 
dimensional image with respect to what the object can be and where it could have been 
positioned and the subject invention utilizes known dimensions with regard to the camera system 
and viewing plane, in combination with assumed characteristics of the piece of paper (column 7, 
lines 28 - 53), which reads on the claimed rendering of the three-dimensional image data model 
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has substantially the pose of the moving model plane and the claimed creating a three 
dimensional image model of a product in a pose corresponding to the pose of the moving model 
plane, wherein orientation of the markers determines the orientation of the three dimensional 
image data model and the claimed display the augmented reality video. Harrington discloses 
video capture hardware that converts the analog signal from the camera into a digitized frame 
that can be stored in the computer and the hardware outputs a signal to the computer as a 
digitized and processible representation of what the user sees from the camera's field of view 
and the video generation hardware takes a bit map of pixel values from the computer and 
converts them into a television format that can be displayed on the headset and where the signal 
generated by the camera is different from the signal displayed on the headset by the addition of 
the computer-generated electronic image to thereby form a synthetic image as an augmented- 
reality display (column 4, lines 25 - 38), which reads on the claimed converting the augmented 
reality video into a streaming video format and the claimed streaming the augmented reality 
video having the streaming video format over a communications network to a computer for 
displaying the augmented reality video. Harrington did not expressly discloses encoding 
hyperlink information into the augmented reality video and accessing information about the 
corresponding product by selecting the corresponding product in the rendered three-dimensional 
form from the augmented reality video. Rhoads discloses watermark may carry information or 
programmatic action or link to external information or an action, such as retrieval and output of 
information stored elsewhere in a database, website, etc (column 4, lines 31-39) and watermark 
can be embedded immediately into a video object layer after a graphical model is rendered to the 
video object layer (column 4, lines 40 - 49) and users selecting a video object, retrieving the 
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linked information or actions for the selected object and rendering them on its user interface 
(column 16, lines 16-38). It would have been obvious for one of ordinary skill in the art at the 
time of the invention to further modify Harrington's system by embedding hyperlink information 
into 3D rendered objects when selected. One would be motivated to do so because this gives 
users the convenience of retrieving extra information about the selected desired object from an 
external source upon request. 

Claims 50, 51, 58, and 59 are rejected under 35 U.S.C 103(a) as being unpatentable over 
Kato et al. ("Marker Tracking and HMD Calibration for a Video-based Augmented Reality 
Conferencing System", 20 October 1999) in view of Rhoads et al. (US 7,050,603). 

Regarding dependent claims 50, 51, 58, and 59, Kato did not expressly discloses 
encoding hyperlink information into the augmented reality video and accessing information 
about the corresponding product by selecting the corresponding product in the rendered three- 
dimensional form from the augmented reality video. Rhoads discloses watermark may carry 
information or programmatic action or link to external information or an action, such as retrieval 
and output of information stored elsewhere in a database, website, etc (column 4, lines 3 1 - 39) 
and watermark can be embedded immediately into a video object layer after a graphical model is 
rendered to the video object layer (column 4, lines 40 - 49) and users selecting a video object, 
retrieving the linked information or actions for the selected object and rendering them on its user 
interface (column 16, lines 16 - 38). It would have been obvious for one of ordinary skill in the 
art at the time of the invention to further modify Kato's system by embedding hyperlink 
information into 3D rendered objects when selected. One would be motivated to do so because 
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this gives users the convenience of retrieving extra information about the selected desired object 
from an external source upon request. 

Conclusion 

Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffrey J. Chow whose telephone number is (571)-272-8078. The 
examiner can normally be reached on Monday - Friday 10:00AM - 5:00PM (EST). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka Chauhan can be reached on (571)-272-7782. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 





